Critical sample size for the Lp-norm estimator in linear regression models

In the presence of non-Gaussian noise the least squares estimator for the parameters of a regression model can be suboptimal. Therefore, it is reasonable to consider other norms. Lp-norm estimators are a useful alternative, particularly when the residuals are heavy-tailed. We analyze the convergence properties of such estimators as a function of the number samples available for estimation. An analysis based on the Random Energy Model (REM), a simplified model used to describe the thermodynamic properties of amorphous solids, shows that, in a specific limit, a second order phase transition takes place: For small sample sizes the typical and average values of the estimator are very different. For sufficiently large samples, the most probable value of the estimator is close to its expected value. The validity analysis is illustrated in the problem of predicting intervals between subsequent tweets.